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United  States 

General  Accounting  Office 

Washington,  D.C.  20548 


Information  Management  and 
Technology  Division 

B-252539 

May  17, 1993 

The  Honorable  Ronald  V.  Dellums 
Chairman,  Committee  on  Armed  Services 
House  of  Representatives 

Dear  Mr.  Chairman: 

This  report  responds  to  your  request  that  we  review  the  investment  strategy  of  the  Advanced 
Research  Projects  Agency’s  (arpa)  High  Performance  Computing  (hpg)  Program.  Specifically, 
you  asked  that  we  assess  (1)  arpa’s  distribution  of  advanced  computers  to  research  sites, 

(2)  arpa’s  interaction  vdth  the  research  community,  and  (3)  the  balance  between  software  and 
hardware  investments  in  the  arpa  program.  The  report  identifies  weaknesses  in  the  program, 
which  could  slow  technological  progress  and  prevent  arpa  from  achieving  its  hpc  program 
goals,  and  makes  recommendations  to  the  Secretary  of  Defense  to  improve  the  program. 

We  will  give  copies  of  the  report  to  appropriate  congressional  committees,  the  Secretary  of 
Defense,  and  other  interested  parties.  Copies  will  also  be  made  available  to  others  upon  request. 

This  work  was  performed  under  the  direction  of  Samuel  W.  Bowlin,  Director,  Defense  and 
Security  Information  Systems,  who  can  be  reached  at  (202)  512-6240.  Other  m^or  contributors 
are  listed  in  appendix  IV. 

Sincerely  yours. 


C' 


Ralph  V.  Carlone 
Assistant  Comptroller  General 


Executive  Summary 


Purpose 


High-performance  computing  refers  to  the  use  of  advanced  computing 
technologies,  especially  supercomputers,  to  solve  highly  complex, 
numerically  intensive  problems  in  the  shortest  possible  time.  These 
scientific  problems — such  as  understanding  global  climate  change  or 
analyzing  molecular  structure — are  collectively  called  the  grand 
challenges.  The  federal  High  Performance  Computing  and 
Communications  Initiative  is  a  research  and  development  effort  that  seeks 
to  significantly  accelerate  the  availability  and  utilization  of  high 
performance  computers  and  networks  in  order  to  better  address  these 
challenges.  At  $275  million  in  fiscal  year  1993,  the  Advanced  Research 
Projects  Agency  (arpa)  has  the  largest  budget  of  any  single  agency 
participating  in  the  $800-million  federal  initiative.* 

Given  the  importance  of  this  initiative  and  arpa’s  dominant  role  in  it,  the 
House  Armed  Services  Committee  asked  gao  to  assess  the  program, 
particularly  (1)  the  agency’s  distribution  of  advanced  computers  to 
research  sites,  (2)  its  interaction  with  the  research  community,  and  (3)  the 
balance  between  hardware  and  software  investments  in  the  arpa  program. 


Background 


ARPA  has  been  funding  high-performance  computing  research  and 
development  since  it  began  a  strategic  computing  program  in  the  early 
1980s,  and  is  now  one  of  the  lead  agencies  in  the  federal  initiative.  It  is  the 
primary  agency  involved  in  the  research  and  development  of  critical 
high-performance  computing  technology,  needed  to  address  the  grand 
challenges,  arpa  coordinates  its  projects  with  other  agencies,  especially 
the  National  Science  Foundation,  which  focus  more  on  applications, 
infrastructure,  and  education,  arpa  funds  some  200  projects  in  this  area  of 
inquiry,  half  managed  by  industry  and  half  by  universities. 

ARPA  has  concentrated  on  a  new  approach  to  supercomputer  design,  based 
on  the  interconnection  of  hundreds  or  even  thousands  of  microprocessors; 
this  is  commonly  known  as  massively  parallel  processing.  Although  arpa’s 
achievements  in  computing  technology  are  widely  recognized,  its  program 
in  this  specific  area  has  been  controversial  because  of  its  seemingly 
narrow  emphasis  on  increasing  machine  speed  and  because  of  its 
continuing  support  for  only  a  few  select  vendors.  The  high-performance 
computing  program  is  also  unusual  for  the  agency  in  that  its  goals  go 
beyond  basic  research,  stressing  development  of  useful  massively  parallel 
processing  systems. 


'Until  March  1993,  ARPA  was  known  as  the  Defense  Advanced  Research  Projects  Agency. 
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Results  in  Brief 


While  ARPA  has  fostered  significant  advances  in  high-performance 
computing  research  and  development  for  a  decade,  its  current  program 
has  shortcomings  in  several  areas.  First,  its  placing  of  new  computers  at 
laboratories,  while  important  to  facilitating  research,  has  focused  on  just  a 
few  massively  parallel  processing  machines.  Researchers  need  access  to  a 
broader  range  of  new  computing  technologies  to  explore  all  promising 
alternatives.  Second,  the  agency’s  limited  interactions  with  the  technical 
community  may  hinder  the  rapid  progress  needed  to  achieve  its  ambitious 
program  goals,  arpa  has  been  weak  in  disseminating  program  information, 
soliciting  input  from  the  research  community,  and  publishing  performance 
data.  Finally,  while  much  progress  has  been  made  in  hardware 
development,  software  remains  too  primitive  to  make  massively  parallel 
processing  systems  useful,  arpa’s  goal  of  achieving  a  thousand-fold 
increase  in  useful  computing  power  by  1996  will  likely  not  be  met  without 
greater  emphasis  on  the  development  of  system  software.^ 


Principal  Findings 


ARPA  Has  Been  Placing 
Too  Narrow  a  Range  of 
Computers 


ARPA  has  actively  sought  to  place  new  massively  parallel  processing 
machines  that  it  has  helped  to  develop  into  the  hands  of  researchers  as 
quickly  as  possible.  This  practice  of  placing  early  prototype  machines  in 
research  settings  is  widely  supported  in  theory,  yet  the  agency’s  specific 
actions  have  been  heavily  criticized  as  biased  toward  the  products  of  two 
vendors  who  have  received  research  and  development  funding  from  the 
agency. 

To  date,  arpa’s  high-performance  computing  and  strategic  computing 
programs  have  facilitated  the  procurement  of  computers  made  by  its 
development  contractors,  including  44  systems  made  by  Intel  Corporation, 
and  24  systems  made  by  Thinking  Machines  Corporation,  as  well  as  a 
number  of  computers  made  by  companies  that  are  no  longer  marketing 
massively  parallel  processing  products.  However,  arpa  has  not  been 
involved  in  any  major  procurement  of  new  machines  made  by  current 
major  rivals  to  Intel  and  Thinking  Machines.  Since  it  is  important  that  a 
broad  variety  of  massively  parallel  processing  designs  be  made  available  to 


^System  software  is  the  collection  of  programs  and  data  that  make  up  and  relate  to  the  operating 
system,  for  example,  input/output  routines,  command-line  interpreters,  and  task  scheduling  and 
memory  management  routines.  Application  software  is  software  designed  to  fulfill  the  specific  needs 
of  a  user. 
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researchers  so  that  all  promising  approaches  can  be  tested,  arpa’s  placing 
of  only  the  products  of  its  development  contractors  is  not  justified. 


Limited  Interaction  With  arpa’s  relatively  large  high-performance  computing  budget  and  past 
Research  Community  computing  accomplishments  make  it  a  leader  in  massively  parallel 

Hinders  Progress  processing  research  and  development.  Within  the  federal  initiative,  the 

agency’s  role  is  also  dominant;  other  agencies  look  to  ARPA-sponsored 
research  and  development  for  technological  advances  that  will  support 
their  own  efforts.  Rapid  progress  in  high-performance  computing  hinges 
on  arpa’s  effectively  interacting  with  the  broad  spectrum  of  researchers 
from  government  laboratories,  academia,  and  industry.  Such  interactions 
with  this  community  have,  however,  been  limited. 

ARPA  gets  input  from  the  research  community  chiefly  by  interacting  with  its 
own  principal  investigators.  It  publishes  no  detailed  summaries  or 
progress  reports.  Researchers  likewise  find  it  difficult  to  understand  how 
ARPA  selects  projects  for  funding.  At  the  other  end  of  the  process, 
individual  agency-sponsored  projects  are  not  required  to  publish  results  in 
any  standard  format  or  at  any  predetermined  time.  The  lack  of  widely 
available  performance  data  on  new  designs  slows  technological  progress 
because  the  research  community  remains  uncertain  about  the  merits  of 
the  new  designs. 


ARPA  Has  Not  Sufficiently 
Addressed  System 
Software 


High-performance  computing  experts  from  academia,  government,  and 
industry  have  criticized  the  agency  for  overemphasizing  hardware. 
Specific  system  software  areas  needing  greater  attention  have  been 
identified,  including  programming  languages,  compilers,  and  program 
development  tools.^  According  to  these  experts,  arpa  must  focus  more  on 
system  software  development  to  meet  its  program  goals. 


'The  development  of  system  software  that  would  enable  researchers  to 
make  full  use  of  the  tremendous  processing  power  of  massively  parallel 
processing  machines  has  not  kept  pace  with  the  development  of  hardware. 
This  shortcoming  threatens  attainment  of  arpa’s  goal  of  achieving  a 
thousand-fold  increase  in  useful  computing  power  by  1996. 


’’Compilers  are  system  software  programs  that  translate  source  code,  written  in  high-level 
programming  languages,  into  machine-executable  object  code. 


Page  4 


GAO/IMTEC-93-24  High  Performance  Computing 


Executive  Summary 


Recommendations 


Agency  Comments 


In  order  to  broaden  participation  in  arpa’s  program  and  to  better  facilitate 
research  into  high-performance  computing,  gao  recommends  that  the 
Secretary  of  Defense  direct  the  Director,  arpa  to  (1)  broaden  the  agency’s 
computer  placement  program  by  including  a  wider  range  of  computers 
from  more  vendors;  (2)  establish  and  maintain  a  public  database  of 
information  about  the  status  and  results  of  the  agency’s  ongoing 
high-performance  computing  projects,  as  well  as  performance  data  for 
different  massively  parallel  processing  systems;  and  (3)  emphasize  and 
support  research  and  development  of  system  software  as  a  major  element 
of  the  agency’s  high-performance  computing  program.  Other  related 
recommendations  are  included  in  chapters  2  through  4  of  the  report. 


GAO  did  not  obtain  written  agency  comments  on  a  draft  of  this  report. 
However,  gao  provided  a  draft  of  the  report  to  arpa  and  discussed  the 
report’s  findings  and  recommendations  with  arpa  officials,  including  the 
Director,  Deputy  Director  for  Management,  and  High  Performance 
Computing  Program  Director.  Their  specific  comments  have  been 
incorporated  in  the  text  and  revisions  to  the  report  have  been  nrade  as 
appropriate.  The  arpa  officials  generally  agreed  that  improvements  could 
be  made  in  the  management  of  the  high-performance  computing  program. 
Regarding  placement  of  computer  systems  at  research  sites,  the  arpa 
officials  noted  that  they  are  already  taking  some  steps  to  broaden  their 
program.  With  respect  to  gao’s  finding  that  arpa’s  interaction  with 
researchers  has  been  limited,  the  arpa  officials  stated  that  their 
dissemination  of  public  information  is  better  than  the  report  indicates. 
Regarding  gao’s  finding  that  the  agency  has  not  sufficiently  addressed 
system  software,  arpa  officials  stated  that  they  have  invested  substantially 
in  software  development.  In  each  of  these  cases,  gao  believes  that  arpa’s 
actions  have  not  been  sufficient  to  invalidate  the  findings  of  the  report. 
The  steps  arpa  is  taking  to  broaden  its  computer  placement  program  are 
still  in  the  early  stages;  gao  believes  attention  to  this  issue  is  stiU  needed. 
arpa’s  efforts  to  disseminate  public  information  still  lack  a  mechanism  for 
providing  detailed  performance  results  from  technology  development 
projects.  Provisions  for  obtaining  broad  scientific  input  to  the  program  are 
also  still  lacking.  Finally,  notwithstanding  arpa’s  investment  to  date  in 
software,  gao  found  during  this  review  that  a  substantially  greater 
emphasis  needs  to  be  placed  on  system  software  development  if  arpa’s 
program  goals  are  to  be  met.  arpa  officials’  oral  comments  and  gao’s 
evaluation  of  them  are  included  at  the  end  of  each  chapter. 
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Introduction 


The  term  high-performance  computing  (hpc)  refers  to  the  use  of  advanced 
computing,  communications,  and  information  technologies,  such  as 
supercomputers  and  high-speed  networks,  to  solve  highly  complex, 
mathematically  intensive  problems  in  the  shortest  possible  time,  hpc  is 
widely  considered  to  be  a  critical  technology  area  that  can  further 
progress  in  solving  fundamental  problems  that  are  critical  to  ensuring 
competitiveness  and  that  require  significant  increases  in  computational 
capability.  Examples  include  prediction  of  global  climate  change, 
determination  of  molecular  structure,  and  understanding  the  nature  of 
new  synthetic  materials.  Some  of  these  examples  are  shown  in  figure  1.1. 
Because  hpc  is  so  important,  the  federal  government  plays  a  leading  role  in 
its  development. 


The  Govemmentwide 
Initiative 


The  federal  High  Performance  Computing  and  Communications  (hpcc) 
Initiative  began  in  fiscal  year  1992  as  a  joint  effort  among  nine  federal 
agencies  to  significantly  accelerate  the  availability  and  utilization  of  the 
next  generation  of  high-performance  computers  and  networks. 
Specifically,  the  initiative  aims  for  a  thousand-fold  improvement  in  useful 
computing  capability  and  a  hundred-fold  improvement  in  available 
computer  communications  capability  by  1996.  There  are  four  major 
components  of  the  hpcc  initiative: 


•  High  Performance  Computing  Systems  (hpcs):  the  development  of  the 
underlying  technology  required  to  build  systems  capable  of  sustaining 
trillions  of  operations  per  second  on  large  problems; 

•  Advanced  Software  Technology  and  Algorithms  (asta):  the  development  of 
software  technology  and  algorithms  to  address  grand  challenge  problems; 

•  National  Research  and  Education  Network  (nren):  the  development  of  a 
national  high-speed  network  to  provide  distributed  computing  capability 
to  research  and  educational  institutions;  and 

•  Basic  Research  and  Human  Resources  (brhr):  support  for  research  by 
individual  investigators  and  initiation  of  activities  to  increase  the  pool  of 
trained  personnel. 


Planned  fiscal  year  1993  spending  by  agency  and  hpcc  component  is 
shown  in  table  1.1. 
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Table  1.1:  HPCC  Initiative  Fiscal  Year 
1993  Budget  by  Agency  and 

Agency 

HPCS 

ASIA 

NREN 

BRHR 

TOTAL 

Component 

ARPA 

119.5 

49.7 

43.6 

62.2 

275.0 

NSF 

25.8 

107.6 

41.1 

50.6 

225.1 

DOE 

10.1 

63.8 

13.8 

12.9 

100.6 

NASA 

12.0 

57.6 

9.1 

3.1 

81.8 

NSA® 

36.2 

5.9 

3.2 

0.2 

45.5 

NIH 

3.0 

31.4 

4.1 

8.0 

46.5 

NOAA 

0.0 

9.4 

0.4 

0.0 

9.8 

EPA 

0.0 

6.0 

0.4 

1.5 

7.9 

Education 

0.0 

0.0 

2.0 

0.0 

2.0 

NIST 

0.3 

0.6 

1.2 

0.0 

2.1 

Total 

206.9 

332.0 

118.9 

138.5 

796.3 

®NSA  joined  the  HPCC  Initiative  in  1993.  Previously  it  had  been  represented  by  ARPA. 


No  single  federal  agency  has  overall  responsibility  for  the  initiative. 
Instead,  participating  agencies  coordinate  their  program  plans  and 
activities  through  an  arm  of  the  White  House  Office  of  Science  and 
Technology  Policy,  which  establishes  task  groups,  prepares  budgets,  and 
coordinates  program  planning  and  execution.  The  four  agencies  involved 
in  preparing  the  original  hpcc  plan — ^the  Department  of  Energy  (doe),  the 
Advanced  Research  Projects  Agency  (arpa),  the  National  Aeronautics  and 
Space  Administration  (nasa),  and  the  National  Science  Foundation 
(nsf) — are  the  major  participants  in  the  initiative. 
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Figure  1.1:  Examples  of  High  Performance  Computing  Applications 


Source:  University  of  Minnesota 


A  Thinking  Machines  CM-5  was  used  for  this 
study  of  fluid  flow  between  two  concentric 
cylinders. 


Source:  Intel  Corporation 


An  Intel  iPSC/860  was  used  by  researchers 
at  Argonne  National  Laboratory  to  study  the 
molecular  structure  of  potential  enzyme 
designs. 


Source:  nCUBE  Corporation 


Researchers  at  Sandia  National  Laboratories 
used  an  nCUBE  MPP  to  model  airflow  as  the 
space  shuttle  reenters  the  Earth’s  atmosphere 
at  8  times  the  speed  of  sound. 


Source:  Intel  Corporation 

Researchers  used  the  Intel  Delta  at  the  California  Institute  of 
Technology  for  this  new  approach  to  global  climate  modeling. 
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Source:  NASA  Goddard  Space  Flight  Center 

Use  of  a  MasPar  MPP  allowed  scientists  to  discern  a  binary  star  formation  (on  right)  that 
was  hidden  in  the  original  (on  left)  taken  by  the  Hubble  Space  Telescope. 


Source:  Lawrence  Livermore  National  Laboratory 

This  study  of  density  fluctuations  in  a  gaseous  structure  used  for 
nuclear  fusion  was  performed  on  a  Thinking  Machines  CM-2. 
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area’s  HPC  Program 


ARPA  is  the  lead  agency  within  the  Department  of  Defense  (dod)  for 
advanced  technology  research,  and  arpa  represents  dod’s  interests  within 
the  federal  hpcc  initiative.  In  general,  arpa  is  tasked  with  pursuing 
imaginative  and  innovative  research  and  development  projects  that 
promise  significant  military  utility,  arpa’s  purview  includes  basic  and 
applied  research  and  development  projects  that  demonstrate  the  feasibility 
of  revolutionary  technological  approaches. 


ARPA  has  a  long  history  of  championing  key  developments  in  the  evolution 
of  computer  technology.  For  example,  the  agency  was  instrumental  in 
developing  a  computer  communications  networking  approach  known  as 
packet  switching  during  the  1960s  and  1970s.  When  arpa  first  began 
building  a  packet-switched  computer  network,  the  technique  was 
unproven;  established  commercial  firms  were  not  interested  in  financing 
packet-switching  research  and  development  on  their  own.  However,  due  in 
large  part  to  the  success  of  arpa’s  initiative,  the  technique  eventually  came 
into  widespread  use  and  has  facilitated  the  interconnection  of 
heterogeneous  computer  systems  and  resources  around  the  world,  arpa 
has  also  been  involved  in  the  development  of  other  important 
technologies,  such  as  advanced  interactive  computer  graphics  and  various 
aspects  of  artificial  intelligence. 

arpa’s  Computing  Systems  Technology  Office  administers  the  agency’s  hpc 
program.  The  program  consists  of  a  number  of  research  and  development 
projects  that  are  carried  out  by  both  academic  researchers  and 
commercial  developers.  Projects  range  from  small-scale  experiments 
involving  new  hardware  and  software  design  concepts  to  large-scale 
development  of  complete  computer  systems.  Currently,  arpa  funds  some 
200  projects  in  hpc,  which  are  split  fairly  evenly  between  industry  and 
university-sponsored  projects.  At  $275  million  in  fiscal  year  1993,  arpa 
controls  the  single  largest  share  of  hpcc  resources. 

Officials  from  various  agencies  involved  in  the  hpcc  initiative  agree  that 
ARPA  is  the  de  facto  leader  in  supporting  research  and  development  of  new 
HPC  technology.  The  agency  has  been  supporting  hpc  research  and 
development  since  the  early  1980s,  when  it  initiated  the  Strategic 
Computing  Program.  One  of  the  Strategic  Computing  Program’s  original 
goals  was  to  expand  practical  experience  with  experimental  parallel 
computer  systems.  In  the  1980s,  arpa’s  support  was  critical  to  the 
development  of  several  massively  parallel  processing  (mpp)  computers, 
including  the  Connection  Machine  and  the  Intel  Touchstone  series.  Prior 
to  this,  parallel  processing  had  been  largely  ignored  by  the  scientific 
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research  community,  and  few  manufacturers  had  offered  mpp  products. 
ARPA  has  maintained  a  steady  investment  in  mpp  research  since  that  time 
and  has  largely  succeeded  in  proving  the  overall  concept  and  feasibility  of 
the  MPP  approach.  The  early  success  of  arpa’s  parallel  computing  research 
led  ARPA  officials  to  become  heavily  involved  in  the  formulation  of  the 
federal  hpcc  initiative. 


Massively  Parallel 
Processing 


MPP  represents  a  revolutionary  new  computer  design.  Most  traditional 
computers  have  one  computational  processor,  and  traditional  computer 
development  has  focused  on  making  this  processor  faster  and  more 
efficient.  However,  the  potential  for  continued  increases  in  speed  is 
reaching  the  limits  imposed  by  the  physical  properties  of  the  materials 
used  to  build  the  processor. 

Most  supercomputers  currently  used  to  solve  “real-world”  problems  are 
vector  supercomputers.  Vector  machines  achieve  much  greater  speeds 
than  traditional  computers  by  employing  processors  that  process  data  in 
an  assembly-line  fashion.  In  a  vector  processing  machine,  a  number  of 
instructions  are  in  various  stages  of  being  processed  at  any  given  moment 
as  they  make  their  way  through  the  machine’s  high-speed  processing 
pipelines.  Results  emerge  at  the  end  of  these  pipelines  just  as  completed 
automobiles  emerge  from  automotive  assembly  lines.  Computers 
manufactured  by  Cray  Research,  Inc.,  are  typical  of  this  kind  of 
supercomputer.  While  these  machines  are  much  more  powerful  than 
traditional  single-processor,  general-purpose  computers,  they  are  still  not 
powerful  enough  to  adequately  address  grand  challenge  problems. 

Many  computer  scientists  believe  that  mpp  promises  speeds  far  surpassing 
those  of  vector  supercomputers  by  breaking  computational  problems  into 
many  separate  parts  and  having  a  large  number  of  processors  tackle  those 
parts  simultaneously.  Speed  is  achieved  largely  through  the  sheer  number 
of  processors  operating  simultaneously,  rather  than  through  any 
exceptional  power  in  each  processor.  In  fact,  many  mpp  designs  use 
commercial,  off-the-shelf  processors,  such  as  those  found  in  personal 
computers  or  scientific  workstations,  and  may  include  hundreds  or  even 
thousands  of  these  processors. 

Most  MPP  designs  are  intended  to  be  scalable;  that  is,  the  machines 
function  effectively  in  configurations  that  range  from  a  small  number  of 
processors  to  a  very  large  number  of  processors.  While  the  number  of 
processors  may  vary,  the  system’s  basic  architecture  and  system  software 
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are  constant.  Thus  these  machines  can  be  tailored  to  match  a  wide  variety 
of  computing  demands. 

The  concept  of  massive  parallelism  can  be  implemented  in  many  different 
ways.  Efficient  methods  must  be  developed  to  break  up  large 
computational  problems  and  assign  tasks  to  individual  processors. 
Likewise,  new  methods  must  be  devised  for  efficiently  managing 
communications  among  the  processors.  Although  some  solutions  to  these 
problems  seem  clearly  better  than  others,  there  is  still  no  consensus  about 
what  approaches  are  best  overall.  Getting  large  amounts  of  data  quickly 
and  efficiently  sent  to  and  retrieved  from  a  large  collection  of  processors 
and  their  associated  memory  modules  also  presents  significant 
engineering  challenges. 


ARPA’s  Goals  Are 
Ambitious 


The  goal  of  arpa’s  hpc  program  is  to  develop  systems  capable  of  sustaining 
trillions  of  operations  per  second  on  grand  challenge  problems  by  1996. 
Current  vector  supercomputers  offer  billions  of  operations  per  second  of 
useful  computing  power;  the  ARPA-sponsored  systems  would  thus 
represent  a  thousand-fold  improvement  in  computational  capability,  arpa’s 
objective  is  not  only  to  achieve  very  high  processing  speeds  in  laboratory 
tests,  but  to  make  that  computing  power  useful  in  addressing  complex 
real-world  problems. 

arpa’s  goals  are  ambitious  because  its  officials  envision  a  substantial 
conversion  of  scientific  research  from  vector  to  massively  parallel 
supercomputing.  However,  much  of  the  theoretical  improvement  in  speed 
and  processing  efficiency  expected  of  mpp  machines  has  yet  to  be  realized. 
Current  mpp  machines  have  demonstrated  superiority  to  vector 
supercomputers  on  only  a  few  applications.  In  order  for  mpp  to  gain 
widespread  acceptance  in  the  scientific  research  community,  its 
effectiveness  on  large,  complex  scientific  problems  will  have  to  be 
demonstrated. 


The  MPP  Research 
and  Development 
Community 


Typically,  mpp  computers  have  been  manufactured  by  small  companies  or 
corporate  divisions  established  specifically  to  research,  develop,  and 
produce  a  particular  mpp  design.  Examples  include  Intel  Supercomputer 
Systems  Division,  Kendall  Square  Research  Corporation,  MasPar 
Computer  Corporation,  Meiko  Scientific  Corporation,  nCUBE 
Corporation,  Parsytec  GmbH,  and  'Thinking  Machines  Corporation. 
Because  the  market  for  these  computers  is  very  limited,  the  few 
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companies  that  are  actively  marketing  products  compete  vigorously  for 
each  sale.  Larger,  more  established  computer  manufacturers  have  been 
slower  to  enter  the  mpp  field  but  are  becoming  more  interested  as  the 
technology  matures.  Elxamples  of  large  computer  manufacturers  that  are 
now  actively  involved  in  mpp  include  Cray  Research,  Incorporated,  Digital 
Equipment  Corporation,  and  International  Business  Machines, 
Incorporated. 

Although  some  mpp  machines  are  in  operation,  achieving  such  goals  as  a 
thousand-fold  increase  in  useful  computing  is  still  a  subject  of  research 
and  development.  Within  the  government,  mpp  research  and  development 
are  conducted  at  doe  and  dod  laboratories,  at  nasa  centers,  and  at  nsf 
supercomputer  centers.  Typically,  these  government  sites  have  extensive 
experience  with  traditional  vector  supercomputers  and  support  large 
numbers  of  users.  Because  these  sites  will  eventually  shift  their  users  to 
MPP  machines,  many  of  them  operate  small  laboratories  where  researchers 
evaluate  how  well  new  mpp  machines  handle  problems  that  typify  their 
interests. 
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ARPA  has  been  actively  involved  in  providing  researchers  with  quick  access 
to  early  prototypes  of  mpp  machines  that  it  has  helped  to  develop.  By 
testing  the  new  machines  on  real-world  problems,  researchers  can  rapidly 
identify  the  machine’s  strengths  and  weaknesses  and  provide  feedback  to 
system  developers.  While  this  activity  is  widely  supported  in  concept, 
arpa’s  specific  actions  have  been  criticized  as  too  narrowly  focused  on  the 
products  of  vendors  who  have  received  research  and  development  funding 
from  ARPA.  To  date,  arpa  has  facilitated  the  acquisition  of  a  relatively  large 
number  of  computers  made  by  Intel  Supercomputing  Systems  Division  and 
Thinking  Machines  Corporation,  but  has  not  provided  to  researchers 
machines  made  by  m^or  rivals  to  these  companies,  including  Kendall 
Square  Research  Corporation,  nCUBE  Corporation,  and  MasPar  Computer 
Corporation.  Some  researchers  and  vendors  have  charged  that  arpa’s  wide 
placement  of  Intel  and  Thinking  Machines  products  has  made  it  difficult 
for  researchers  to  gain  access  to  rival  machines.  By  making  a  broader 
selection  of  alternative  mpp  designs  available,  arpa  would  not  only  dispel 
this  controversy  but  also  better  ensure  that  promising  mpp  design 
alternatives  are  not  overlooked. 


The  Need  for  Rapid 
Deployment  of  New 
Computers 


Research  and  development  in  the  field  of  massively  parallel  processing  is 
progressing  at  a  rapid  pace.  Members  of  the  mpp  community,  including 
government  agencies,  laboratories,  and  vendors,  vigorously  compete  with 
each  other  to  be  at  the  forefront  of  the  field.  In  order  for  rapid  progress  to 
occur,  new  mpp  machines  must  not  only  be  built  but  must  also  be  made 
accessible  to  researchers  and  thoroughly  tested  so  that  their  strengths  and 
weaknesses  can  be  identified.  The  results  of  this  experimental  testing  can 
then  be  used  to  influence  future  versions  of  the  machines. 

If  new  machines  are  built  but  not  provided  to  the  community,  real-world 
experiments  do  not  occur  and  significant  technical  progress  cannot  be 
made.  In  such  cases,  the  government  has  funded  a  laboratory  curiosity — a 
single  prototype  that  few  researchers  can  access  and  that  contributes  little 
to  the  body  of  engineering  design  experience  in  the  field. 

Researchers  generally  view  the  government’s  standard  acquisition  process 
for  computer  equipment  as  a  barrier  to  effective  participation  in  hpc 
research.  According  to  these  researchers,  standard  goverrunent 
procurement  procedures  for  acquiring  mzyor  computer  systems  can  take 
as  many  as  2  to  3  years  to  complete,  by  which  time  a  new  generation  of 
MPP  machines  will  have  been  produced.  If  no  alternative  acquisition 
process  were  available,  government  laboratories  could  not  effectively 
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participate  in  hpc  research.  Because  some  agencies  have  no  special 
procedures  for  such  time-criticail,  research-oriented  procurements,  arpa 
assists  them,  using  special  procedures  for  rapidly  acquiring  advanced 
computer  prototypes. 


ARPA’s  Program 


ARPA  works  closely  with  industry  and  academia  to  develop  new 
technologies,  such  as  mpp.  However,  it  does  not  operate  any  laboratories  of 
its  own  where  these  technologies  can  be  tested.  Instead,  arpa  forms 
partnerships  with  other  government  agencies,  laboratories,  and  industry  to 
evaluate  the  technology  it  helps  to  develop,  arpa  places  computer  systems 
at  sites  controlled  by  these  other  organizations  based  on  proposals  it 
receives  from  them  or  as  part  of  joint  projects  with  other  government 
agencies.  Systems  placed  at  government  and  academic  sites  may  be  paid 
for  by  either  arpa  or  the  site,  and  arpa’s  involvement  may  bring  a  lower 
price,  since  arpa  has  ongoing  contractual  relationships  with  some  mpp 
developers.  The  site  is  often  responsible  for  operations  and  maintenance 
expenses. 

Almost  all  of  the  mpp  systems  that  have  been  placed  so  far  by  arpa  were 
manufactured  by  either  Intel  or  Thinking  Machines.  ‘  To  date,  arpa’s  hpc 
and  Strategic  Computing  programs  have  facilitated  the  procurement  of  44 
computer  systems  made  by  Intel  and  24  systems  made  by  Thinking 
Machines.  Appendix  II  contains  complete  lists  of  these  placements, 
including  the  sites  where  the  machines  were  placed,  their  cost,  and  the 
source  of  the  funding,  arpa  supported  the  development  of  both  vendors’ 
machines.  As  part  of  the  development  contracts  for  these  machines,  arpa 
negotiated  options  that  would  allow  the  government  to  buy  a  certain 
number  of  “early  prototype  copies”  at  a  discount  for  evaluation  purposes. 


Industry  and 
Researchers  Have 
Questioned  ARPA’s 
Placement  of  MPP 
Machines 


The  fact  that  arpa  is  able  to  provide  machines  quickly  to  the  research 
community  is  something  that  many  researchers  value.  Nevertheless,  much 
controversy  surrounds  arpa’s  placement  of  particular  mpp  machines.  A 
number  of  government  officials,  researchers,  and  vendors  have  said  that 
arpa’s  actions  have  distorted  the  market  for  mpp  machines  because  arpa 
has  almost  exclusively  favored  the  products  of  just  two  companies,  Intel 
and  Thinking  Machines.  The  arpa  hpc  program  has  not  supported  any 
procurements  of  machines  made  by  several  other  mgjor  mpp 
developers — such  as  Kendall  Square  Research  Corporation,  nCUBE 


'In  the  late  1980s,  ARPA  also  placed  machines  made  by  vendors  who  have  since  left  the  MPP  field, 
including  BBN  Advanced  Computers,  Inc.  and  Encore  Computer  Company. 
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Corporation,  and  MasPar  Computer  Corporation — with  whom  arpa  did  not 
have  MPP  development  contracts. 

Industiy  representatives  are  concerned  that  the  Intel  and  Thinking 
Machines  products  placed  by  arpa  largely  fall  into  the  category  of 
platforms  for  research  into  the  applied  use  of  mpp  instead  of  early 
prototypes  to  be  rigorously  tested  for  engineering  soundness.  Researchers 
and  industry  representatives  whom  we  interviewed  generally  agreed  that 
the  placement  of  just  two  or  three  machines  at  key  research  and 
development  sites  would  be  sufficient  for  rapid  testing  of  early  prototypes. 
ARPA  has  placed  many  more  Touchstones,  iWarps,  and  Connection 
Machines  than  this.  Researchers  argue  that  all  eligible  mpp  machines 
should  be  involved  in  the  placement  program  if  the  emphasis  is  on  applied 
research  rather  than  early  prototype  testing. 

While  some  of  the  placements  of  the  Intel  and  Thinking  Machines  products 
were  clearly  intended  for  early  testing  of  prototype  versions,  other 
placements  have  been  used  largely  for  research  into  the  applicability  of 
parallel  processing  to  various  scientific  disciplines.  For  example,  two  of 
the  Intel  machines,  a  Touchstone  and  an  iWarp,  were  placed  by  arpa  at  the 
Naval  Ocean  Systems  Center’s  Center  for  Advanced  Computation.  Navy 
documents  indicate  that  the  primary  purpose  of  the  center  is  to  explore 
new  MPP  applications,  such  as  anti-submarine  warfare,  not  to  advance 
computer  science  per  se.  Other  sites  also  fit  the  same  profile. 

Design  and  engineering  information  about  the  Intel  and  Thinking  Machines 
products  also  indicates  that  later  editions  of  the  machines  acquired 
through  ARPA  were  significantly  more  refined  than  the  first  few  machines 
produced  by  each  company.  Intel’s  first  two  deliveries  under  the 
Touchstone  program,  to  nasa  Ames  Research  Center  and  Oak  Ridge 
National  Laboratory,  were  clearly  early  prototypes  intended  for  field 
testing.  The  machines  were  marked  “Touchstone  Gamma,”  the  official 
designation  of  the  prototype  program,  and,  according  to  Intel  officials, 
were  significantly  less  refined  from  an  engineering  standpoint  than  later 
copies  of  the  machines.  The  additional  Touchstone  machines  placed  by 
ARPA  were  all  marked  “iPSC/860” — Intel’s  commercial  designation  for  the 
machine — and  were  engineered  to  production  standards.  In  the  case  of 
Thinking  Machines,  copies  of  the  CM-2  and  CM-200  machines  placed  by 
ARPA  were  essentially  refinements  of  the  original  CM-1  design.  They  did  not 
represent  any  significant  design  departure  from  the  earlier  machine. 
However,  arpa  facilitated  a  number  of  CM-2  and  CM-200  acquisitions. 
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Researchers  and  industry  representatives  are  also  concerned  that  arpa’s 
wide  placement  of  Intel  and  Thinking  Machines  products  has  made  it 
difficult  to  get  access  to  alternative  mpp  computers.  For  example,  since 
officials  at  government  laboratories  are  aware  that  arpa  has  a  program  to 
place  Intel  and  Thinking  Machines  products  in  laboratories,  they  may  be 
more  inclined  to  take  advantage  of  this  opportunity  than  to  independently 
acquire  alternative  machines.  Even  researchers  who  are  simply  looking  for 
access  to  existing  mpp  machines  have  found  it  much  easier  to  get  access  to 
Intel  and  Thinking  Machines  computers  than  other  mpp  machines  because 
the  ARPA-sponsored  products  have  been  widely  placed  in  the  community. 
Accordingly,  the  research  community  may  be  less  able  to  fully  explore 
alternative  designs  because  a  lack  of  arpa  support  has  limited  the  variety 
of  MPP  machines  available  at  laboratories. 

Recognizing  the  concerns  regarding  its  narrow  placement  of  mpp 
computers,  arpa  has  begun  to  experiment  with  an  alternative  approach.  In 
a  January  1992  solicitation,  arpa  offered  to  include  qualified  hpc 
developers  who  do  not  already  have  development  contracts  with  arpa  in 
its  prototype  placement  program.^  However,  contracts  stenuning  from  this 
provision  do  not  specifically  guarantee  sales  of  any  particular  vendor’s 
products  but  merely  make  a  vendor  eligible  for  consideration.  Although 
ARPA  is  currently  negotiating  contracts  related  to  this  provision  with 
several  vendors,  no  computer  placements  have  yet  been  made. 


Conclusions 


It  is  important  that  mechanisms  be  in  place  to  provide  new,  advanced 
computers  to  the  research  community  in  a  rapid,  streamlined  fashion,  arpa 
has  been  addressing  that  need,  but  only  with  machines  it  has  helped  to 
develop.  Since  it  appears  that  at  least  some  of  the  arpa  placements  of 
machines  were  primarily  for  research  into  the  applicability  of  mpp  rather 
than  primarily  for  rapid  feedback  on  the  design  of  the  machines,  arpa  does 
not  appear  to  be  justified  in  restricting  the  program  to  only  those 
machines  that  it  helped  to  develop.  A  broader  and  richer  range  of 
experimental  results  could  be  obtained  by  placing  a  wider  range  of 
machine  designs,  arpa  has  already  begun  experimenting  with  a  more 
broadly  based  approach  to  placement  of  machines,  arpa’s  goal  of 
stimulating  the  aggressive  advance  of  the  high-performance  computing 
technology  base  would  be  best  served  by  adopting  such  an  approach  for 
aU  of  its  mpp  placements. 


^Broad  Agency  Announcement  92-07,  “Research  in  High  Performance  Computing  Systems,”  Commerce 
Business  Daily,  Jan.  27, 1992. 
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Recommendations 

We  recommend  that  the  Secretary  of  Defense  direct  the  Director,  arpa,  to 
(1)  broaden  the  agency’s  computer  placement  program  by  including  a 
wider  range  of  computers  from  more  vendors;  (2)  ensure  that  the  agency’s 
decisions  to  support  high-performance  computing  research  projects  are 
not  influenced  by  whether  ARPA-supported  computing  platforms  have  been 
specified  in  researchers’  proposals;  and  (3)  develop,  maintain,  and  publish 
a  comprehensive  list  of  qualified  machines,  including  both  ARPA-sponsored 
and  non-ARPA-sponsored  designs,  that  arpa  will  assist  sponsored 
researchers  to  acquire. 

Agency  Comments 


ARPA  officials  stressed  that  they  are  already  taking  steps  to  broaden  their 
placement  of  mpp  machines.  Specifically,  they  said  that  the  alternative 
approach  to  placing  computer  systems  that  they  are  experimenting  with 
represents  significant  progress  in  broadening  the  range  of  machines  the 
agency  places.  As  discussed  in  the  report,  we  recognize  that  arpa  is 
currently  negotiating  contracts  related  to  this  provision  with  a  number  of 
vendors.  However,  no  computer  placements  have  yet  been  made.  We 
believe  that  attention  to  this  issue  is  still  needed  to  ensure  that  a  broader 
program  ensues. 


Also,  ARPA  officials  stated  that  the  agency  has  recently  been  involved  in  a 
joint  project  with  nsf  to  place  computer  equipment  made  by  a  current  rival 
to  Intel  and  Thinking  Machines.  Specifically,  the  agency  funded 
enhancements  to  two  existing  computer  systems  manufactured  by  Kendall 
Square  Research  Corporation,  which  are  located  at  Cornell  University  and 
the  Oak  Ridge  National  Laboratory.  However,  in  our  opinion,  these 
enhancements  are  not  comparable  to  the  placement  of  complete  new 
systems.  The  totals  that  we  report  for  Intel  and  Thinking  Machines  were 
for  the  purchase  and  installation  of  new  machines,  not  upgrades  or 
enhancements  to  existing  installations. 
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Despite  its  widely  recognized  leadership  role  in  hpc  research  and 
development,  arpa’s  program  remains  isolated  from  the  larger  hpcc 
research  community.  Little  information  is  published  about  the  direction  of 
the  program  or  the  projects  that  are  being  funded,  leaving  potential 
participants  uncertain  about  arpa’s  specific  technical  priorities,  arpa’s 
process  for  reviewing  and  selecting  proposals  is  also  somewhat  informal 
and  difficult  for  observers  to  follow.  Since  project  results  are  not  required 
to  be  reported  at  any  predetermined  time,  up-to-date  performance  data  on 
new  MPP  designs  developed  by  arpa  have  been  hard  to  obtain.  The  lack  of 
such  data  hinders  wider  adoption  of  mpp  technology  because  the  scientific 
communily  remains  uncertain  about  the  merits  of  new  designs.  To 
stimulate  rapid  hpc  progress,  arpa  will  need  to  interact  more  effectively 
vuth  the  broader  hpcc  community  and  disseminate  information  about  the 
objectives  and  results  of  its  program  expeditiously,  including  timely 
performance  data  on  new  mpp  designs. 


ARPA’s  Interaction 
With  the  Research 
Community  Is  Too 
Limited 


Although  ARPA  funding  plays  a  key  role  in  the  hpc  community,  the  agency 
obtains  little  input  from  the  community  on  the  direction  of  its  program  and 
disseminates  little  public  information  about  its  projects  and  their  status. 
HPC  researchers  without  arpa  contracts  have  limited  opportunity  for  input 
into  the  technical  direction  of  the  arpa  program.  Program  managers  within 
the  agency  formulate  hpc  goals  and  priorities  based  on  their  knowledge  of 
the  field.  According  to  arpa  officials,  the  chief  mechanism  for  obtaining 
input  on  the  overall  direction  of  the  hpc  program  is  through  discussions 
with  ARPA-sponsored  researchers  at  principal  investigator  meetings. 
Forums  that  would  allow  for  direct  input  from  non-ARPA  sponsored 
researchers  do  not  exist,  and  attendance  at  principal  investigator  meetings 
has  been  criticized  by  researchers  as  not  representative  of  key  research 
and  development  constituents  who  have  significant  stakes  in  the  overall 
HPCC  program.  A  more  open  meeting,  such  as  an  annual  or  semi-annual 
conference  on  the  direction  of  the  hpc  program,  would  allow  for  broader 
community  involvement  and  promote  sharing  of  research  experience. 

ARPA  has  also  disseminated  little  public  documentation  on  the  hpc  projects 
it  funds  and  the  rationale  for  this  funding,  arpa  formerly  issued  annual 
reports  for  its  Strategic  Computing  Program  that  provided  overviews  of 
the  projects  being  funded,  budget  allocations,  and  results  obtained  to  date. 
However,  these  annual  reports,  which  were  issued  from  1985  through 
1988,  have  been  discontinued,  arpa  officials  said  that  they  decided  to 
discontinue  the  annual  reports  because  researchers  were  using  the  reports 
as  guidance  in  preparing  new  proposals,  arpa  wanted  researchers  instead 
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to  see  the  program  as  a  “clean  slate,”  open  to  any  creative  and  original 
proposal.  However,  the  resulting  lack  of  public  information  has  led  to 
confusion  among  potential  bidders  about  the  specific  direction  of  arpa’s 
program. 

Potential  bidders  have  also  had  trouble  learning  the  outcome  of  arpa’s 
project  selection  process.*  Specifically,  arpa  does  not  do  enough  to  notify 
bidders  of  results  in  a  timely  and  orderly  way.  A  dozen  or  more  proposals 
may  be  selected  from  the  results  of  a  single  broad  agency  announcement 
(baa),  and  these  are  turned  over  to  a  variety  of  government  contracting 
agencies  to  be  separately  negotiated.  Because  each  contracting  agency 
then  proceeds  at  its  own  pace,  there  is  no  convenient,  single  point  in  time 
when  all  the  contracts  resulting  from  a  given  baa  are  announced  together. 

Furthermore,  arpa  does  not  attempt  to  give  a  consolidated  public 
accounting  of  the  results  of  each  baa.  As  a  result,  it  is  difficult  for  bidders 
whose  proposals  are  rejected  to  learn  what  projects  were  selected  in  their 
place  or  why.  This  lack  of  information,  in  turn,  reinforces  perceptions 
among  some  members  of  the  mpp  community  that  arpa  favors  certain 
bidders  and  is  unwilling  to  work  with  others.  Better  information 
dissemination  about  baa  selection  results  could  encourage  wider 
participation  and  could  allay  questions  about  arpa’s  fairness. 


Project  Results  Are 
Not  Published  in  a 
Uniform  and  Timely 
Fashion 


ARPA  does  not  require  its  researchers  to  report  on  the  results  of  their 
projects  in  any  standard  format  or  at  any  predetermined  time.  Although 
researchers  often  publish  papers  and  articles  describing  project  results, 
there  is  no  coordinated  source  for  information  on  the  status  of  ongoing 
HPC  projects.  There  may  be  no  way  to  obtain  performance  data  on  a  new 
system  supported  by  arpa  until  perhaps  a  year  or  more  after  the  new 
machine  has  been  introduced.  This  results  in  uncertainty  about  the 
technical  merits  of  new  designs  and  adds  risk  to  mpp  procurement 
decisionmaking  at  government  laboratories. 

A  case  in  point  is  the  Connection  Machine  CM-5,  manufactured  by 
Thinking  Machines  Corporation.  The  CM-5  was  a  m^or  new  mpp  product 
that  attracted  attention  in  the  research  community.  Although  its  design 
theoretically  promised  significantly  greater  processing  speed  than  older 
MPP  machines,  actual  performance  was  hard  to  predict  in  the  abstract. 
Without  quantitative  measurement  of  the  machine’s  performance  running 


'Appendix  III  contains  a  full  analysis  of  ARPA’s  HPC  project  selection  process  and  discusses  the 
results  of  a  sample  case. 
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real  applications,  researchers  said  they  had  difficulty  determining  the 
CM-5’s  true  capabilities.  For  example,  one  nasa  official  involved  in 
planning  mpp  research  said  that  he  had  heard  comments  about  the  CM-5’s 
performance  ranging  from  “terrific”  to  “atrocious”  but  had  seen  no 
quantitative  data  to  support  either  claim.  Another  noted  that  the  lack  of 
performance  data  on  the  machine  had  initially  made  researchers  skeptical 
about  the  CM-5’s  capabilities.  Although  many  researchers  consider  the 
CM-5  to  be  a  significant  advance  in  mpp  development,  quantitative 
performance  data  still  remain  scarce. 


MPP  Performance 
Measurement 
Standards  Are 
Lacking 


There  are  no  widely  accepted  methods  for  measuring  and  comparing  the 
performance  of  different  mpp  systems.  Although  a  variety  of  benchmark 
problem  sets  have  been  developed  that  typify  the  problem  domains  of 
specific  scientific  and  engineering  disciplines,  these  benchmarks  are  often 
only  useful  within  those  disciplines.  As  a  result,  the  performance  of  these 
machines  in  other  domains  is  unknown,  which  makes  it  difficult  for 
researchers  who  are  interested  in  applying  mpp  to  their  work  to  gauge 
progress  in  the  field. 

ARPA  does  not  support  the  use  of  universal  benchmarks  to  evaluate  and 
compare  the  performance  of  mpp  systems,  arguing  that  the  difference 
among  mpp  designs  is  too  great  for  a  single  suite  of  benchmark  tests  to 
adequately  profile,  arpa  officials  believe  that  potential  mpp  users  should 
instead  test  machines  by  using  ARPA-sponsored  libraries  of  software 
modules  developed  for  their  disciplines,  arpa  argues  that  trying  out  these 
software  modules  would  give  prospective  users  a  good  idea  of  how  weU  a 
given  machine  would  perform.  While  this  could  theoretically  take  the 
place  of  benchmarks  for  some  users,  a  mechanism  is  still  needed  to 
compile  comparable  performance  results  across  platforms  and  make  them 
readily  available.  Advocates  of  such  benchmarks  maintain  that  a  standard 
performance  metric  is  needed  to  force  an  objective  assessment  of  the 
relative  merits  of  different  mpp  systems.  They  argue  that  such  an 
assessment  would  allow  potential  mpp  users  to  make  more  intelligent 
investment  choices  and  would  focus  mpp  research  and  development  efforts 
on  overcoming  performance  shortfalls. 

Even  if  standard  performance  metrics  remain  difficult  to  establish,  arpa 
could  nevertheless  improve  the  amount  and  quality  of  performance  data 
that  are  available  to  the  research  commimity.  Some  experts  have 
suggested  that  a  consolidated  database  be  established  that  would  offer 
researchers  easy  access  to  information  about  the  status  and  results  of 
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ongoing  hpcc  projects,  including  performance  assessments  of  new 
supercomputer  designs.  Establishing  such  a  database  would  go  far  to 
resolve  arpa’s  information  dissemination  shortcomings.  In  addition  to 
providing  potential  users  of  mpp  technology  with  data  on  which  to  base 
their  investment  decisions,  such  a  database  would  mark  progress  in  both 
hardware  and  software  toward  the  hpc  program’s  goals  and  provide 
feedback  to  the  conununity  on  research  results  and  areas  in  need  of 
emphasis  or  redirection. 

Conclusions 

m  —  — - 

arpa’s  leadership  role  in  hpg  research  and  development  carries  with  it  the 
responsibility  to  address  the  needs  of  researchers  as  broadly  and  as  fairly 
as  possible.  Unaccustomed  to  dealing  with  such  a  broad  constituency, 

ARPA  has  focused  on  interacting  chiefly  with  its  own  principal 
investigators.  As  a  result,  the  agency  has  not  devoted  enough  attention  to 
disseminating  information  about  its  hpc  program  and  its  project  selection 
process  to  the  larger  hpcc  community,  arpa  needs  to  widen  its  contact 
with  this  community,  disseminating  information  more  expeditiously  and 
soliciting  more  input  regarding  the  direction  of  its  program.  Establishing  a 
database  of  performance  and  program  data  would  go  far  to  resolve  the 
agency’s  information  dissemination  shortcomings. 

Recommendations 

We  recommend  that  the  Secretary  of  Defense  direct  the  Director,  arpa,  to 
(1)  establish  and  maintain  a  public  database  of  information  about  the 
status  and  results  of  the  agency’s  ongoing  hpc  projects  as  well  as 
performance  data  for  different  mpp  systems,  and  (2)  sponsor  annual  or 
semi-annual  conferences  on  the  direction  of  the  hpc  program  specifically 
to  provide  a  forum  for  broad  scientific  input  to  the  agency’s  hpc  program. 

Agency  Comments 

■  ^  - - 

ARPA  officials  said  they  are  considering  alternatives  for  improving 

information  dissemination  within  the  hpc  program,  such  as  establishing  a 
public  database  of  program  information.  However,  they  stated  that  they  do 
not  consider  their  current  information  dissemination  activity  to  be  lacking. 
In  support  of  their  contention,  they  referred  to  several  public  conferences 
on  supercomputing  that  have  been  held  where  general  information  about 
the  govemmentwide  hpcc  program — ^including  arpa’s  component — ^was 
disseminated.  They  also  cited  published  materials  associated  with  their 
principal  investigator  meetings.  However,  we  remain  concerned  that  the 
amount  and  level  of  detail  of  the  information  provided  through  these 
mechanisms  is  inadequate.  Information  about  the  scope  and  direction  of 
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the  ARPA  program  was  presented  at  the  public  conferences  at  only  a  very 
abbreviated  and  generalized  level.  Likewise,  the  published  results  of  arpa 
principal  investigator  meetings  have  consisted  merely  of  collections  of 
presentation  graphics,  with  no  explanatory  material  that  would  make  them 
useful  to  individuals  who  were  unable  to  attend  the  meetings. 

Furthermore,  none  of  these  mechanisms  for  information  dissemination 
addresses  the  need  to  obtain  broad  scientific  input  to  the  agency’s  hpc 
program.  Nor  does  any  mechanism  offer  a  way  to  ensure  that  detailed 
performance  results  from  critical  new  technology  projects  are 
disseminated  expeditiously. 
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ARPA  must  address  critical  software  shortcomings  quickly  in  order  to  meet 
its  HPC  program  goal  of  making  mpp  useful  in  addressing  a  variety  of 
complex  real-world  problems.  Although  hardware  systems  capable  of 
scaling  up  to  1  trillion  operations  per  second  appear  to  be  achievable  by 
1996,  it  is  much  less  likely  that  the  system  software  required  for  users  to 
harness  this  power  will  be  available  at  that  time.  System  software 
development  has  not  kept  pace  with  hardware  development,  leaving 
current  systems  difficult  to  use.  User  groups  studying  this  problem  have 
identified  mpp  programming  languages,  advanced  compiler  systems,  and 
program  development  tools  as  specific  areas  that  are  critical  to  making 
MPP  systems  more  useful.  Unless  arpa  shifts  the  balance  of  its  program  to 
emphasize  these  software  areas,  it  appears  unlikely  that  the  capabilities  of 
advanced  mpp  supercomputers  will  be  fully  exploitable  by  1996. 


Experts  Have 
Emphasized  the  Need 
for  Greater 
Investment  in  System 
Software 


There  is  widespread  concern  among  university  and  industry  experts  that 
software  shortcomings  are  holding  back  further  advances  in  mpp.  These 
experts  argue  that  mpp  will  not  gain  widespread  acceptance  unless  rapid 
progress  is  made  in  developing  system  software.  Such  progress  can  only 
be  achieved  by  investing  more  heavily  in  all  phases  of  hpc  software 
research  and  development. 

Three  separate  conferences  convened  in  1992  to  identify  critical  directions 
in  research  and  development  for  hpc  have  emphasized  the  critical  role  of 
system  software  in  ensuring  the  success  of  the  hpcc  program.  The  Purdue 
Workshop  on  Grand  Challenges  in  Computer  Architecture  for  the  Support 
of  High  Performance  Computing,  sponsored  by  the  National  Science 
Foundation,  stressed  that  a  substantial  effort  must  be  devoted  to 
advancing  mpp  system  software. 


Similarly,  the  Pasadena  Workshop  on  System  Software  and  Tools  for  High 
Performance  Computing  Environments,  sponsored  by  a  number  of  federal 
agencies  and  attended  by  over  a  hundred  experts  from  industry,  academia 
and  government,  concluded  that  a  substantial  investment  of  resources  and 
time  in  the  research  and  development  of  system  software  will  be  required 
to  make  mpp  hardware  systems  useful.  Scientists  participating  in  an 
Industry  Advisory  Board  forum,  convened  by  International  Business 
Machines  and  the  Association  for  Computing  Machinery,  also  concluded 
that  the  lack  of  system  software  is  the  most  serious  obstacle  confronting 
MPP  users. 
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Primitive  System 
Software  Limits  MPP’s 
Usefulness 


For  most  potential  mpp  users,  the  high  cost  of  developing  software  for  their 
specific  applications  still  outweighs  the  benefits  of  greatly  improved 
processing  speed  associated  with  mpp.  Application  software  development 
is  difficult  because  current  mpp  systems  are  equipped  with  only  rather 
primitive  system  software,  thus  requiring  extra  effort  on  the  part  of 
application  programmers.  For  example,  mpp  programmers  usually  must  be 
highly  versed  in  the  particular  architectural  features  of  their  machines, 
such  as  details  about  how  the  memory  is  configured.  In  many  cases,  they 
also  must  specify  exactly  how  information  controlled  by  different 
processors  should  be  exchanged.  Furthermore,  once  they  have  written 
their  programs,  few  tools  are  available  to  assist  programmers  in  examining 
how  the  instructions  are  divided  among  and  executed  by  the  processors  so 
that  errors  can  be  isolated  and  performance  improved.  As  a  result,  only 
the  most  highly  skilled  potential  users  can  effectively  exploit  the 
performance  capabilities  of  most  mpp  systems.  In  addition  to  the 
immediate  negative  impact  on  scientific  researchers,  the  lack  of  system 
software  also  discourages  vendors  from  developing  commercial  mpp 
application  software,  because  such  products  may  only  work  on  a  few 
specific  hardware  platforms  that  may  quickly  become  obsolete. 


There  is  general  consensus  in  the  hpc  community  that  several  system 
software  technologies  must  be  researched  and  developed  soon  in  order  to 
ease  the  burden  on  programmers  and  accelerate  conversion  to  mpp.  These 
technologies  include:  (1)  parallel  programming  languages  and  compilers 
and  (2)  software  development  tools. 


Better  Parallel 
Programming  Languages 
and  Compilers  Are  Needed 


Programmers  are  often  unable  to  take  full  advantage  of  the  capabilities  of 
MPP  machines  because  the  programming  languages  and  compilers  available 
today  do  not  have  adequate  functionality.  Although  development  of 
software  tends  to  lag  behind  hardware  development  for  all  types  of 
computer  systems,  certain  factors  have  made  development  of  parallel 
programming  languages  and  compilers  especially  difficult.  Since  mpp 
designs  are  widely  divergent,  each  with  its  unique  organization  of  multiple 
processors  and  memory  hierarchies,  it  is  especially  difficult  to  develop 
system  software  that  can  efficiently  interact  with  each  design.  As  a  result, 
most  current  mpp  computers  have  relatively  weak  system  software  that 
does  not  perform  all  the  functions  that  it  should.  For  example,  mpp 
programmers  must  explicitly  synchronize  and  schedule  concurrent 
program  activities  and  resources,  a  complex  and  arcane  task  that 
consumes  a  great  deal  of  programming  effort  and  is  often  beyond  the 
experience  of  most  application  programmers. 
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Better  Software 
Development  Tools  Are 
Needed 


Existing  parallel  programming  languages  and  compilers  generally  cannot 
carry  out  important  functions  efficiently  and  effectively  or  differ  widely  in 
the  approaches  they  take.  Examples  of  such  functions  include  (1)  how  a 
user  controls  parallel  input  and  output,  (2)  the  way  in  which  processors 
that  are  working  in  parallel  on  a  problem  communicate  among  themselves, 
and  (3)  how  data  are  to  be  arranged  in  memory. 

Computer  scientists  acknowledge  that  much  work  needs  to  be  done  to 
develop  languages  and  compilers  that  would  automatically  address  all 
these  unique  mpp  functions  in  a  highly  efficient  and  effective  way,  and  they 
agree  that  this  is  an  area  that  should  dominate  future  research  and 
development  efforts. 


Parallel  program  debuggers  are  foremost  among  the  tools  that  are  needed 
to  enable  users  to  develop  mpp  application  software  efficiently.  Parallel 
programs  are  more  difficult  to  debug  than  traditional  programs  because  of 
the  many  processors  involved,  each  of  which  may  independently 
encounter  programming  errors  while  other  processors  are  executing 
software  instructions  correctly.  The  problem  is  complicated  by  the  fact 
that  erroneous  results  from  one  processor  are  likely  to  be  fed  into 
computations  performed  by  other  processors.  The  improper  timing  of 
activities  shared  among  several  processors  can  cause  errors  that  are  hard 
to  detect  because  they  do  not  occur  in  the  same  way  every  time  the 
program  is  run.  Debugging  facilities  capable  of  diagnosing  these 
timing-dependent  errors  and  relating  them  back  to  the  programmer’s  code 
have  not  yet  been  developed.  The  lack  of  such  tools  severely  hinders  mpp 
users’  efforts  to  design  and  develop  parallel  applications. 

Even  if  a  parallel  program  is  free  of  errors,  it  may  not  run  as  quickly  or 
efficiently  as  the  programmer  expects.  The  problem  may  be  due  to  the 
user’s  inexperience  in  allocating  parallel  hardware  resources  to  computing 
tasks.  Tools  that  enable  users  to  interact  with  and  understand  the  behavior 
of  their  programs  are  essential  for  producing  programs  that  make  the  most 
efficient  use  of  mpp  hardware  resources.  Preferably,  these  tools  would 
provide  a  graphic  analysis  and  explanation  so  that  the  programmer  could 
visualize  the  problem  and  more  readily  correct  it. 


Page  28 


GAO/IMTEC-93-24  High  Performance  Computing 


Chapter  4 

ARPA  Must  Emphasize  Software  to  Meet  Its 
Goals 


ARPA’s  Approach 
Focuses  on  Early 
Phases  of  Software 
Development 


ARPA  officials  recognize  that  software  issues  must  be  resolved  in  order  to 
reach  their  goal  of  enabling  the  application  of  mpp  to  the  grand  challenges 
by  1996.  They  stress  that  the  objective  of  the  arpa  hpc  program  is  to 
develop  mpp  systems,  including  hardware  and  software.  The  arpa  program 
includes  a  system  software  component  intended  to  enhance  the  practical 
usefulness  of  mpp  systems.  Compilers,  programming  languages,  and 
software  development  tools  are  all  being  addressed  to  some  degree  within 
this  program. 

Nevertheless,  users  with  whom  we  spoke  were  almost  unanimous  in 
criticizing  arpa’s  support  for  parallel  software  research  and  development 
as  inadequate.  Although  arpa  is  involved  in  software  research  and 
development,  its  commitment  to  making  the  same  kind  of  advances  in 
software  as  it  has  made  in  hardware  is  not  apparent  to  researchers.  This 
may  be  in  part  because  arpa’s  strategy  for  software  research  and 
development  is  to  invest  most  in  early  work  that  explores  new  software 
concepts  and  involves  experimentation  with  laboratory  models,  arpa’s 
policy  is  not  to  spend  heavily  on  the  later,  advanced  stages  of  software 
development,  which  concentrate  on  making  reliable  products  for 
widespread  use. 

Experts  working  on  mpp  software  say  that  arpa’s  strategy  is  inadequate  for 
the  current  state  of  the  mpp  industry.  They  maintain  that  arpa  needs  to 
emphasize  software  development  by  continuing  to  fund  the  most 
promising  of  the  research  prototypes  through  the  advanced  development 
stage.  Readying  more  software  products  for  evaluation  by  end  users  is 
seen  as  the  best  way  to  accelerate  the  community’s  acceptance  of  mpp. 
Furthermore,  commercial  software  developers,  who  heretofore  have  been 
reluctant  to  invest  heavily  in  the  development  of  mpp  software,  would 
likely  be  encouraged  to  produce  more  refined  commercial  software 
products.  This  is  because  arpa  support  for  any  given  technology  is  often 
seen  by  commercial  developers  as  leading  to  greater  acceptance  and  more 
widespread  utilization  by  the  user  community. 


Conclusions 


Current  mpp  software  shortcomings  are  significant.  Experts  in  the  field 
agree  that  these  shortcomings  are  a  barrier  to  effective  utilization  of  mpp 
within  the  scientific  community.  Specific  software  areas  in  need  of 
intensive  near-term  research  and  development  have  been  identified  by 
expert  panels.  However  arpa  has  not  provided  strong  leadership  nor  has  it 
made  a  clear  commitment  to  accelerating  system  software  research  and 
development  efforts.  If  arpa  is  to  meet  its  goal  of  enabling  the  effective 
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application  of  mpp  to  grand  challenge  problems  by  1996,  it  will  need  to  shift 
its  emphasis  to  software. 

Recommendations 

We  recommend  that  the  Secretary  of  Defense  direct  the  Director,  arpa,  to 
(1)  emphasize  and  support  the  research  and  development  of  mpp  system 
software  as  a  major  element  of  the  agency’s  hpc  program,  and  (2)  ensure 
that  a  significant  body  of  software  development  work  is  underway  to 
aggressively  advance  the  technology  base,  especially  in  mpp  programming 
languages,  effective  compiler  systems,  and  debugging  and  performance 
analysis  tools.  In  order  to  achieve  broad  mpp  use  by  1996,  arpa  must  fund 
promising  software  prototypes  to  an  advanced  stage  of  development  as  it 
does  for  hardware,  rather  than  limiting  itself  to  supporting  only  the  earlier 
stages  of  research  and  development. 

Agency  Comments 

■i 

ARPA  officials  disagreed  with  our  statement  that  arpa  is  not  sufficiently 
addressing  mpp  system  software.  They  stated  that  they  have  made 
substantial  investments  in  software  development,  including  providing 
funding  to  most  prominent  software  researchers  in  the  field.  We  do  not 
dispute  that  arpa  has  funded  a  substantial  number  of  software 
development  projects.  However,  it  was  clear  from  our  review  that  the  level 
of  system  software  investment  that  has  been  made  heretofore  has  not 
succeeded  in  creating  stable,  reliable  computing  platforms  that  scientists 
can  use  to  address  grand  challenge  problems.  Researchers  we  contacted 
were  in  broad  agreement  that  progress  in  this  area  will  need  to  accelerate 
if  ARPA  is  to  reach  its  objective  of  making  mpp  systems  useful  to 
researchers  by  1996.  Accordingly,  greater  emphasis  on  developing  system 
software  is  still  needed. 
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In  its  report  accompanying  the  National  Defense  Authorization  Act  for 
Fiscal  Year  1993,  the  House  Committee  on  Armed  Services  requested  that 
we  assess  the  investment  strategy  of  the  arpa  hpc  program.  During 
subsequent  meetings  with  committee  staff,  our  specific  objectives  were 
established.  These  objectives  were  to  assess  (1)  arpa’s  distribution  of 
advanced  computers  to  research  sites,  (2)  arpa’s  interaction  with  the 
research  community,  and  (3)  the  balance  between  software  and  hardware 
investments  in  the  arpa  program. 

To  meet  these  objectives,  we  obtained  and  reviewed  official  program 
documentation  and  discussed  these  issues  with  government,  private 
industry,  and  academic  officials  from  a  wide  range  of  organizations. 

Specifically,  with  regard  to  arpa’s  role  within  the  federal  hpcc  program,  we 
interviewed  officials  at: 

•  Office  of  Science  and  Technology  Policy,  Executive  Office  of  the 
President,  Washington,  D.C.; 

•  National  Coordination  Office  for  hpcc.  National  Library  of  Medicine, 
Bethesda,  Maryland; 

•  National  Science  Foundation,  Directorate  for  Computer  and  Infonnation 
Science  and  Engineering,  Washington,  D.C.; 

•  Department  of  Energy,  Office  of  Energy  Research,  Gaithersburg, 

Maryland;  and 

•  Department  of  Commerce,  International  Trade  Administration, 
Washington,  D.C. 

Regarding  the  specifics  of  the  arpa  program,  we  analyzed  summary 
records  of  computer  procurements  and  arpa’s  project  selection  process 
and  interviewed  officials  at: 

•  ARPA,  Computing  Systems  Technology  Office,  Arlington,  Virginia; 

•  Office  of  the  Director  of  Defense  Research  and  Engineering,  Washington, 
D.C.; 

•  National  Research  Council,  Computer  Science  and  Telecommunications 
Board,  Washington,  D.C.;  and 

•  National  Institute  of  Standards  and  Technology,  Computer  Systems 
Laboratory,  Gaithersburg,  Maryland. 

We  also  interviewed  scientific  researchers  and  officials  at  government 
laboratories,  including: 
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•  National  Security  Agency,  Ft.  Meade,  Maryland; 

•  Army  High  Performance  Computing  Research  Center,  Minneapolis, 
Minnesota; 

•  David  Taylor  Research  Center,  Carderock,  Maryland; 

•  Naval  Research  Laboratory,  Washington,  D.C.; 

•  Naval  Command,  Control  and  Ocean  Surveillance  Center,  San  Diego, 
California; 

•  San  Diego  Supercomputer  Center,  San  Diego,  California; 

•  Lawrence  Livermore  National  Laboratory,  Livermore,  California; 

•  Los  Alamos  National  Laboratory,  Los  Alamos,  New  Mexico; 

•  Sandia  National  Laboratory,  Albuquerque,  New  Mexico; 

•  NASA  Ames  Research  Center,  Moffett  Field,  California;  and 

•  NASA  Goddard  Space  Flight  Center,  Greenbelt,  Maryland. 

We  interviewed  representatives  from  the  academic  community  at: 

•  National  Science  Foundation,  Washington,  D.C.; 

•  Stanford  University,  Palo  Alto,  California; 

•  University  of  Maryland,  College  Park,  Maryland; 

•  Massachusetts  Institute  of  Technology,  Cambridge,  Massachusetts;  and 

•  Minnesota  Supercomputer  Center,  Mirmeapolis,  Minnesota. 

We  interviewed  mpp  industry  officials  representing: 

•  Cray  Research,  Inc.; 

•  Intel  Corporation,  Supercomputer  Systems  Division; 

•  Kendall  Square  Research  Corporation; 

•  MasPar  Computer  Corporation; 

•  nCUBE  Corporation;  and 

•  Thinking  Machines  Corporation. 

We  also  reviewed  reports  and  documents  related  to  hpc  prepared  by 
various  advisory  groups  and  committees  and  attended  a  discussion  of  hpc 
organized  by  the  American  Electronics  Association  in  Washington,  D.C. 

We  conducted  our  review  from  June  1992  to  February  1993,  in  accordance 
with  generally  accepted  government  auditing  standards. 
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Table  11.1: 

Intel  Touchstone  Machines  Placed  by  ARPA 

— 

— 

Date 

Site 

Machine 

Type 

Acquired 

Number  of 

Nodes 

Cost 

Source  of 
Funding 

10-30-89 

NASA 

gamma 

new 

128 

$1,534,100 

NASA 

06-07-90 

NSA 

gamma 

new 

16 

$445,949 

NSA 

07-17-90 

NIH 

gamma 

new 

16 

$501,117 

NIH 

07-20-90 

Wright-Patterson  Air  Force  Base 

gamma 

new 

three  8 
one  16 

$620,020 

$518,938 

Air  Force 
Army 

08-03-90 

NASA 

gamma 

new 

two  8 

$399,834 

NASA 

08-20-90 

Naval  Ocean  Systems  Center 

gamma 

new 

16 

$380,000 

Navy 

08-20-90 

NSA 

gamma 

upgrade 

16 

$333,960 

NSA 

12-05-90 

Naval  Coastal  Systems  Center 

gamma 

new 

one  32  one  16 

$840,000 

Navy 

12-10-90 

Stanford  University 

gamma 

new 

32 

$604,300 

ARPA 

03-21-91 

Naval  Research  Laboratory,  Phillips 
Laboratory,  Wright-Patterson  Air  Force 
Base,  Ballistics  Research  Laboratory 

gamma 

new 

seven  8 

$1,333,983 

ARPA 

03-22-91 

San  Diego  Supercomputer  Center, 

Cornell  Theory  Center 

gamma 

new 

two  32 

$1,016,730 

ARPA 

04-01-91 

ARPA 

gamma 

new 

three  8  one  32 

$1,036,500 

ARPA 

04-03-91 

Naval  Ocean  Systems  Center 

gamma 

upgrade 

16 

$83,415 

$145,000 

ARPA 

Navy 

05-30-91 

NSA 

gamma 

upgrade 

32 

$637,849 

NSA 

05-31-91 

NIH 

gamma 

upgrade 

64 

$952,688 

NIH 

06-04-91 

Naval  Research  Laboratory 

gamma 

upgrade 

two  8 

$131,560 

ARPA 

06-06-91 

ARPA 

gamma 

new 

two  8 

$291,895 

ARPA 

08-16-91 

Naval  Coastal  Systems  Center 

gamma 

upgrade 

16 

$231,165 

Navy 

08-16-91 

Naval  Research  Laboratory 

gamma 

upgrade 

8 

$137,595 

Navy 

08-16-91 

Sandia  National  Laboratories 

gamma 

upgrade 

24 

$496,096 

DOE 

09-04-91 

Phillips  Laboratory 

gamma 

upgrade 

8 

$11,325 

Air  Force 

09-11-91 

NASA 

gamma 

upgrade 

16 

$285,655 

NASA 

12-11-91 

NIH 

gamma 

upgrade 

48 

$936,097 

NIH 

12-12-91 

DOE 

gamma 

new 

64 

$995,000 

DOE 

02-28-92 

Naval  Coastal  Systems  Center 

gamma 

upgrade 

combined  two  32 
to  one  64 

$151,860 

Navy 

04-01-92 

NIH 

gamma 

upgrade 

64 

$459,384 

NIH 

05-13-92 

Naval  Research  Laboratory 

gamma 

upgrade 

8 

$121,000 

Navy 

05-20-92 

NSA 

sigma 

new 

60 

$1,111,500 

NSA 

09-01-92 

NASA 

sigma 

new 

28 

$3,367,500 

NASA 

09-11-92 

ARPA 

sigma 

new 

60 

$1,425,936 

ARPA 

Note:  32  Touchstone  prototypes  have  been  placed  by  ARPA,  not  including  upgrades. 
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Table  II.2:  Intel  iWarp  Machines  Placed  by  ARPA 

Date 

Site 

Machine 

Type 

Acquired 

Number  of 

Nodes 

Cost 

Source  of 
Funding 

4-17-90 

Naval  Ocean  Systems  Center 

iWarp 

ne\w 

t\wo64 

$260,000 

$540,000 

ARPA 

Navy 

02-07-91 

Naval  Undersea  Warfare  Center 

iWarp 

new 

32 

$241,380 

ARPA 

03-26-91 

Naval  Undersea  Warfare  Center 

iWarp 

new 

64 

$481,350 

Navy 

04-29-91 

University  of  Maryland 

iWarp 

new 

two  32  one  16 

$471,660 

$143,550 

ARPA 

07-91 

Carnegie  Mellon  University 

iWarp 

new 

three  64 

$1,091,250 

$615,110 

ARPA 

Strategic 

Defense 

Initiative 

Organization 

07-03-91 

NSA 

iWarp 

new 

64 

$420,300 

NSA 

03-03-91 

Naval  Air  Warfare  Center 

iWarp 

new 

64 

$50,240 

$400,000 

ARPA 

Navy 

03-03-92 

Naval  Undersea  Warfare  Center 

iWarp 

upgrade 

32 

$221,700 

Note:  Twelve  iWarp  prototypes  have  been  placed  by  ARPA,  not  including  upgrades.  In  addition, 
ARPA  funded  the  placement  of  16  eight-node  development  systems  at  a  total  cost  of  $2,007,660, 
for  experimentation  and  code  development  by  government  researchers  and  contractors. 
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Table  11.3:  Thinking  Machines  Corp,  Connection  Machines  Placed  by  ARPA 

Machine  Number  of  Source  of 


Date 

Site 

Type 

Acquired 

Nodes 

Cost 

Funding 

Network  Server 

CM1 

new 

16,000 

Naval  Research  Laboratory 

CM1 

new 

16,000 

08-29-86 

Science  Applications  International 
Corporation 

CM1 

new 

16,000 

$2,820,000 

ARPA 

Syracuse  University 

CM1 

new 

16,000 

Lockheed  Corporation 

CM2 

new 

two  16,000 

Syracuse  University 

CM2 

new 

16,000 

06-24-87 

SRI  International 

CM2 

new 

8,000 

$3,3S0,000 

ARPA 

The  Analytical  Sciences  Corporation 

CM2 

new 

8,000 

University  of  Maryland 

CM2 

new 

16,000 

05-16-88 

Naval  Research  Laboratory 

CM2 

new 

16,000 

$2,483,113 

2,072,202 

ARPA 

Strategic 

Defense 

Initiative 

Organization 

University  of  Southern  California 

CM2 

new 

16,000 

NASA 

CM2 

new 

16,000 

06-88 

National  Center  for  Atmospheric 
Research 

CM2 

new 

8,000 

$62S,000 

ARPA 

08-88 

Sandia  National  Laboratories 

CM2 

new 

16,000 

$1,109,005 

ARPA 

11-89 

Naval  Ocean  Systems  Center 

CM2 

new 

8,000 

$610,000 

Navy 

Advanced  Decision  Systems 

CM2 

new 

8,000 

11-14-89 

Engineering  Topographic  Laboratory 

CM2 

upgrade 

8,000 

$1,100,000 

ARPA 

Science  Applications  International 
Corporation 

CM2 

upgrade 

8,000 

Naval  Research  Laboratory 

CM2 

upgrade 

8,000 

12-89 

Naval  Undersea  Warfare  Center 

CM2 

new 

8,000 

$835,000 

Navy 

Harvard  University 

CM2 

new 

4,000 

12-18-89 

SRI  International 

CM2 

new 

4,000 

$1,827,000 

ARPA 

Naval  Ocean  Systems  Center 

CM2 

new 

8,000 

05-16-91 

NASA 

CM2 

upgrade 

16,000 

$1,960,338 

NASA 

12-88 

Naval  Research  Laboratory 

CM2 

various  upgrades 

16,000 

$850,443 

Navy 

11-92 

Naval  Research  Laboratory 

CMS 

new 

128 

$100,000 

$4,833,000 

ARPA 

Navy 

12-92 

ARPA 

CM5 

new 

32 

$989,724 

ARPA 

01-93 

NASA 

CMS 

new 

128 

$4,636,740 

NASA 

Note:  Twenty-four  Connection  Machine  prototypes  of  various  sizes  have  been  placed  by  ARPA, 
not  including  upgrades. 
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Once  ARPA  has  identified  the  specific  technical  areas  that  it  intends  to  fund, 
ARPA  formulates  one  or  more  broad  agency  Announcements  (baa),  which 
state  its  intention  of  investing  in  a  particular  field  and  describe  in  general 
terms  its  near-term  research  goals,  the  criteria  for  proposal  selection,  the 
method  of  evaluation,  the  deadline  for  submitting  proposals,  and 
instructions  on  how  proposals  should  be  prepared  and  submitted,  arpa 
then  releases  the  baa  to  the  research  community  by  publishing  it  in  the 
Commerce  Business  Daily,  arpa  baas  may  be  jointly  announced  with  other 
federal  agencies. 
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Figure  111.1:  ARPA*s  HPC  Project  Selection  Process 
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Prior  to  developing  and  subnutting  a  full  proposal,  responding 
organizations  may  submit  an  abstract  proposal  to  get  an  early  indication  of 
whether  their  proposals  are  of  interest  to  arpa.  Teams  of  officials,  largely 
from  ARPA  but  also  including  technical  experts  from  other  agencies,  then 
meet  to  review  the  abstracts.  Feedback  on  the  abstract  proposals  is  in  the 
form  of  a  simple  “yes”  or  “no,”  although  a  “no”  response  does  not  preclude 
later  submission  of  a  full  proposal. 

After  all  proposals  have  been  submitted,  they  are  reviewed  and  grouped 
into  major  technical  categories,  which  teams  of  officials  evaluate  based  on 
arpa’s  technical  priorities  and  the  proposals’  estimated  costs.  The 
reviewers  then  sort  the  proposals  into  three  categories:  selectable, 
potentially  selectable  with  technical  issues  to  resolve,  and  no  potential  to 
negotiate,  arpa  staff  then  attempt  to  resolve  the  unresolved  technical 
issues  by  contacting  the  bidders  and  obtaining  clarification  about  their 
bids.  Once  this  process  is  complete,  the  hpc  Program  Director  and  the 
Director’s  staff  select  proposals  for  funding. 

Following  the  selection  of  the  proposals,  notification  letters  are  sent  to  the 
bidders  stating  that  either  their  proposal  (1)  has  been  selected  for 
negotiation,  (2)  has  potential  for  future  arpa  funding,  or  (3)  was  not 
selected  for  award. 

Program  managers  then  prepare  an  arpa  Procurement  Request  for  selected 
proposals.  This  request  is  a  one-  or  two-page  description  of  the  project  and 
how  it  relates  to  arpa’s  program.  It  also  describes  how  arpa  plans  to  fund 
the  project.  The  procurement  request  is  then  given  to  the  hpc  Program 
Director  for  approval. 

After  this  entire  procedure  is  complete,  arpa  sends  the  approved  proposal 
either  to  arpa’s  Contract  Management  Office  or  to  another  federal  agency, 
which  then  negotiates  the  final  contract  with  the  bidder. 

Under  any  given  baa,  a  variety  of  contracts  ranging  in  scope  and  size  may 
be  signed,  all  at  different  times,  through  a  variety  of  contracting  agencies. 
About  two-thirds  of  the  contracts  arpa  awards  are  drawn  up  by  other 
agencies.  Only  a  few  of  the  larger,  more  complex  contracts  are  directly 
managed  by  arpa’s  Contract  Management  Office,  which  negotiates  the 
contracts  directly  with  bidders,  arpa’s  selection  of  any  given  project  is 
only  made  public  when  a  contract  is  signed.  According  to  arpa  officials, 

95  percent  of  awards  will  be  made  within  1  year  of  the  deadline  for 
proposal  submissions. 
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A  Case  Study: 
Synopsis  #57 


Synopsis  #57,  one  of  the  earliest  baas  issued  within  the  Strategic 
Computing  Program,  was  published  in  the  Commerce  Business  Daily  on 
April  1, 1987,  by  arpa  and  the  Space  and  Naval  Warfare  Systems  Command 
to  solicit  proposals  for  research  in  the  area  of  parallel  computing  systems. 
The  announcement  was  open  for  3  months  after  publication,  arpa  received 
38  proposals  in  response  to  Synopsis  #57.  These  proposals  were  sorted 
into  five  different  technical  categories:  (1)  General  Purpose  Processors, 

(2)  Very  Large-scale  Integration  and  Packaging,  (3)  Processor  Studies  and 
Support,  (4)  Signal  and  Systolic  Processors,  and  (5)  proposals  spanning 
more  than  one  of  the  other  categories.  Proposals  in  the  first  four 
categories  were  evaluated  by  teams  that  weighed  all  the  submissions  in 
each  category  against  each  other.  Proposals  in  the  fifth  category  were 
assessed  individually  by  arpa  officials. 

ARPA  awarded  15  contracts:  two  in  the  General  Purpose  Processor 
category,  three  in  the  Very  Large-scale  Integration  and  Packaging  category, 
three  in  the  Processor  Studies  and  Support  category,  and  none  in  the 
Signal  and  Systolic  Processor  category.  In  addition,  seven  proposals  from 
the  fifth  category  were  selected.  These  contracts  were  awarded  between 
June  1987  and  September  1988  and  had  a  combined  cost  of  about 
$55.7  million.  Table  III.l  lists  the  awardees  and  the  total  amounts  of  the 
contracts. 
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ARPA’s  HPC  Project  Selection  Process 


Table  III.1 :  Proposals  Awarded  Under 
Synopsis  #57 

Awarded  Bidder 

Date 

Contract  Total 

California  Institute  of  Technology 

June  1987 

$4,120,650 

Stanford  University 

August  1987 

$6,443,579 

Massachusetts  Institute  of  Technology 

September  1987 

$6,377,000 

Stanford  University 

December  1987 

$1,493,277 

Syracuse  University 

January  1988 

$4,932,360 

Intel  Corporation 

March  1988 

$9,300,000 

Microelectronics  and  Computer 
Technology  Corporation 

April  1988 

$7,623,544 

University  of  Southern  California 

May  1988 

$5,151,349 

Princeton  University 

July  1988 

$1,496,895 

Massachusetts  Institute  of  Technology 

July  1988 

July  1988 
(two  contracts) 

$3,653,083 

University  of  California,  Santa  Barbara 

July  1988 

$1,329,850 

ESL,  Incorporated 

August  1988 

$1,685,650 

Stanford  University 

August  1988 

$1,853,490 

Syracuse  University 

September  1988 

$279,994 

Note:  Intel’s  Touchstone  project,  which  was  one  of  the  seven  individually  evaluated  proposals, 
was  awarded  under  Synopsis  #57.  It  was  a  36-month  effort  costing  $9.3  million,  the  largest 
contract  ARPA  awarded  under  this  BAA.  ARPA  awarded  the  last  contract  under  this 
announcement  in  September  1988.  All  except  one  of  these  contracts  have  been  completed. 
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Major  Contributors  to  This  Report 


Information 
Management  and 
Technology  Division, 
Washington,  D.C. 


Franklin  W.  Deffer,  Project  Director 
John  A.  de  Ferrari,  Project  Manager 
Dr.  Rona  B.  Stillman,  Chief  Scientist 
Elizabeth  L.  Johnston,  Computer  Scientist 
Nancy  M.  Kamita,  Computer  Scientist 
Lynne  L.  Goldfarb,  Publishing  Adviser 


(610856) 
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